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® (57) Abstract: The invention provides methods of determining a correction for a misalignment. between at least two images in a 
sequence of images due at least in part to sample movement. The methods are applied, for example, in the processing and analysis of 

^ a sequence of images of biological tissue in a diagnostic procedure. The invention also provides methods of validating the correction 
for a misalignment between at least two images in a sequence of images of a sample. The methods may be applied in deciding 

^ whether a correction for misalignment accurately accounts for sample motion. 
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METHODS AND SYSTEMS FOR CORRECTING IMAGE MISALIGNMENT 



5 Prior Applications 

[0001] The present application claims the benefit of U.S. Patent Application Serial Number 
10/273,51 1, filed October 18, 2002, and U.S. Provisional Patent Application Serial Number 
60/414,767, filed on September 30, 2002. 

Field of the Invention 

10 [0002] This invention relates generally to image processing. More particularly, the invention 
relates to correcting image misalignment, where the misalignment is due at least in part to 
sample movement 

Background of the Invention 
[0003] In modern medical practice, it is useful to analyze a sequence of images of in vivo 

15 tissue obtained throughout the course of a diagnostic medical procedure. For example, in 
screening for some forms of cervical cancer, a chemical agent is applied to cervical tissue and 
the optical response of the tissue is captured in a sequence of colposcopic images. The tissue is 
characterized by analyzing the time-dependent response of the tissue, as recorded in the 
sequence of images. During this type of diagnostic procedure, the tissue may move while 

20 images are being taken, resulting in a spatial shift of the tissue within the image frame field. The 
tissue movement may be caused by the natural movement of the patient during the procedure, 
which can occur even though the patient attempts to remain completely still. Accurate analysis 
of the sequence of images may require that the images be adjusted prior to analysis to 
compensate for misalignment caused at least in part by patient movement. 
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5 [0004] There is currently a method of stabilizing an electronic image by generating a motion 
vector which represents the amount and direction of motion occurring between consecutive 
frames of a video signal. See U.S. Patent No. 5,289,274 to Kondo. However, this method 
accounts for certain gross movements of a video camera - in particular, certain vibrations caused 
by the operator of a handheld camcorder. The method does not compensate for misalignment 
10 caused by movement of a sample. For example, such a method could not be used to adequately 
correct an image misalignment caused by the small-scale movement of a patient during a 
diagnostic procedure. 

[0005] Another image stabilization method is based on detecting the physical movement of the 
camera itself. See U.S. Patent No. 5,253,071 to MacKay, which describes the use of a gimbaled 

15 ring assembly that moves as a camera is physically jittered. These types of methods cannot be 
used to correct misalignments caused by the movement of a sample. 

Summary of the Invention 
[0006] The invention provides methods of correcting misalignments between sequential 
images of a sample. The invention is particularly useful for correcting image misalignment due 

20 to movement of the sample between images and/or during image acquisition. The invention also 
allows for real-time, dynamic image alignment for improved optical diagnosis and assessment. 
[0007] In a preferred embodiment, the invention comprises determining an x-displacement and 
a y-displacement corresponding to a misalignment between two images of a tissue sample, where 
the misalignment is caused by a shift in the position of the sample with respect to the image 

25 frame field. For example, in obtaining a sequence of images of an in-situ tissue sample, an 

embodiment of the invention makes it possible to correct for small image misalignments caused 
by unavoidable patient motion, such as motion due to breathing. It has been discovered that 
validating misalignment corrections improves the accuracy of diagnostic procedures that use data 
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5 from sequential images, particularly where the misalignments are small and the need for 

accuracy is great Thus, methods of the invention comprise validating misalignment corrections 
by splitting individual images into smaller subimages, determining displacement between these 
subimages, and comparing the subimage displacements to the overall image displacement. 
Alternatively, validation may comprise adjusting two images according to a misalignment 

10 correction, then determining displacement between corresponding subimages and comparing 
these displacements with a threshold maximum value. 

[0008] It has also been discovered that application of a chemical contrast agent, such as acetic 
acid, prior to or during acquisition of a sequence of tissue images enhances the detection of 
small-scale image misalignment by increasing intra-image contrast of the tissue images. The 
15 enhanced contrast of the tissue features recorded in the images allows for more accurate motion 
correction determination, since enhanced features may serve as landmarks in determining values 
of displacement. 

[0009] Both misalignment correction determination and validation may be performed such that 
an accurate adjustment is made for a misalignment before an entire sequence of images is 

20 obtained. This allows, for example, "on the fly" adjustment of a camera while a diagnostic exam 
is in progress. Thus, corrections may be determined, validated, and accurately adjusted for as 
misalignments occur, reducing the need for retakes and providing immediate feedback as to 
whether an examination is erroneous. Automatic adjustment may be accomplished by adjusting 
aspects of the optical interrogation of the sample using a misalignment correction value. 

25 Adjustments may be performed, for example, by adjusting aspects of transmission and/or 

reception of electromagnetic energy associated with the sample. This may include, for example, 
transmitting a correction signal to a galvanometer system or a voice coil to "null out" a 
misalignment by adjusting the position of a minor or other component of the camera obtaining 
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5 the images according to the correction signal. Alternatively, or additionally, adjustments may be 
performed by electronically adjusting an aspect of an image, for example, the frame and/or 
bounds of an image, according to a misalignment correction value, or by performing any other 
appropriate adjustment procedure. 

[0010] Applications of methods of the invention include the processing and analysis of a 

10 sequence of images of biological tissue. For example, chemical agents are often applied to tissue 
prior to optical measurement in order to elucidate physiological properties of the tissue. In one 
embodiment, acetic acid is applied to cervical tissue in order to whiten the tissue in a way that 
allows enhanced optical discrimination between normal tissue and certain kinds of diseased 
tissue. The acetowhitening technique, as well as other diagnostic techniques, and the analysis of 

15 images and spectral data obtained during acetowhitening tests are described in co-owned U.S. 
patent application Serial No. 10/099,881, filed March 15, 2002, and co-owned U.S. patent 
application entitled, "Method and Apparatus for Identifying Spectral Artifacts," identified by 
Attorney Docket Number MDS-033, filed September 13, 2002, both of which are hereby 
incorporated by reference. 

20 [0011] A typical misalignment between two images is less than about 0.55-mm within a two- 
dimensional, 480 x 500 pixel image frame field covering an area of approximately 25-mm x 25- 
mm. These dimensions provide an example of the relative scale of misalignment versus image 
size. In some instances it is only necessary to compensate for misalignments of less than about 
one millimeter within the exemplary image frame field defined above. In other cases, it is 

25 necessary to compensate for misalignments of less than about 0.3-mm within the exemplary 

image frame field above. Also, the dimensions represented by the image frame field, the number 
of pixels of the image frame field, and/or the pixel resolution may differ from the values shown 
above. 
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5 [0012] A misalignment correction determination may be inaccurate, for example, due to any 
one or a combination of the following: non-translational sample motion such as rotational 
motion, local deformation, and/or warping; changing features of a sample such as whitening of 
tissue; and image recording problems such as focus adjustment, missing images, blurred or 
distorted images, low signal-to-noise ratio, and computational artifacts. Validation procedures of 
10 the invention identify such inaccuracies. The methods of validation may be conducted "on-the- 
fly" in concert with the methods of determining misalignment corrections in order to improve 
accuracy and to reduce the time required to conduct a given test 

[0013] Once an image misalignment is detected, an embodiment provides for automatically 
adjusting an optical signal detection device, such as a camera. For example, a camera may be 
15 adjusted "on-the-fly" to compensate for misalignments as images are obtained. This improves 
accuracy and reduces the time required to conduct a given test. 

[0014] The optical signal detection device comprises a camera, a spectrometer, or any other 
device which detects optical signals. The optical signal may be emitted by the sample, diffusely 
reflected by the sample, transmitted through the sample, or otherwise conveyed from the sample. 
20 The optical signal comprises light of wavelength falling in a range between about 190-nm and 
about 1 100-nm. One embodiment comprises obtaining one or more of the following from one or 
more regions of the tissue sample: fluorescence spectral data, reflectance spectral data, and 
video images. 

[0015] Methods comprise analysis of a sample of human tissue, such as cervical tissue. 
25 Methods of the invention also include analysis of other types of tissue, such as non-cervical 
tissue and/or nonhuman tissue. For example, methods comprise analysis of one or more of the 
following types of tissue: colorectal, gastroesophageal, urinary bladder, lung, skin, and any 
other tissue type comprising epithelial cells. 
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5 [0016] A common source of misalignment is movement of a sample. Methods comprise the 
steps of: obtaining a plurality of sequential images of a sample using an optical signal detection 
device; determining a correction for a misalignment between two or more of the sequential 
images, where the misalignment is due at least in part to a movement of the sample; and 
compensating for the misalignment by automatically adjusting the optical signal detection 
10 device. 

[0017] The two or more sequential images may be consecutive, or they may be 
nonconsecutive. In one embodiment, a misalignment correction is identified between a first 
image and a second image, where the second image is subsequent to the first image. The first 
image and second image may be either consecutive or nonconsecutive. 

15 [0018] Identifying a misalignment correction may involve data filtering. For example, some 
methods comprise filtering a subset of data from a first image of a plurality of sequential images. 
A variety of data filtering techniques may be used. In one embodiment, Laplacian of Gaussian 
filtering is performed. Identifying a misalignment may comprise preprocessing a subset of data 
from the first image prior to filtering. For example, color intensities may be converted to gray 

20 scale before filtering. In some embodiments, filtering comprises frequency domain filtering 
and/or discrete convolution in the space domain. 

[0019] In order to identify a correction for a misalignment, preferred embodiments comprise 
computing a cross correlation using data from each of two of the plurality of sequential images. 
In some embodiments, computing a cross correlation comprises computing a product represented 
25 by Fi(u,v) F*j(u,v), where Fj(u,v) is a Fourier transform of data derived from a subset of data 
from a first image, i, of the plurality of sequential images, F*j(u,v) is a complex conjugate of a 
Fourier transform of data derived from a subset of data from a second image, j, of the plurality of 
sequential images, and u and v are frequency domain variables. In preferred embodiments, the 
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5 computing of the cross correlation additionally comprises computing an inverse Fourier 
transform of the product represented by Fj(u,v) F*j(u,v). 

[0020] A method of the invention comprises validating a correction for a misalignment 
determined between a first image and a second image. Validating a misalignment correction 
comprises defining one or more validation cells within a bounded image plane; computing for 

10 each validation cell a measure of displacement between two (or more) images bound by the 

image plane using data from the two images corresponding to each validation cell; and validating 
a correction for misalignment between the two images by comparing the validation cell 
displacements with the correction. Preferably, each validation cell comprises a subset of the 
bounded image plane. The two (or more) images may be consecutive images. In some 

15 embodiments, the validating step includes eliminating from consideration one or more measures 
of displacement for corresponding validation cells. For example, measures of displacement from 
validation cells determined to be likely to contribute to an erroneous validation result are 
eliminated in some embodiments. In some embodiments, identifying validation cells that are 
likely to contribute to an erroneous validation result comprises calculating a sum squared 

20 gradient for at least one validation cell. 

[0021] Methods of the invention comprise obtaining a plurality of sequential images of the 
sample during an application of a chemical agent to the sample. For example, the chemical 
agent comprises at least one of the following: acetic acid, formic acid, propionic acid, butyric 
acid, LugoPs iodine, Shiller's iodine, methylene blue, toluidine blue, indigo carmine, 

25 indocyanine green, and fluorescein. Some embodiments comprise obtaining sequential images 
of the sample during an acetowhitening test. 
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5 [0022] In preferred embodiments, the movement of the sample is relative to the optical signal 
detection device and comprises at least one of the following: translational motion, rotational 
motion, warping, and local deformation. 

[0023] One or more of the sequential images comprise measurements of an optical signal from 
the sample. The optical signal comprises, for example, visible light, fluoresced light, and/or 

10 another form of electromagnetic radiation. 

[0024] Methods of the invention comprise determining a correction for misalignment between 
each of a plurality of pairs of images. Such methods comprise the steps of: obtaining a set of 
sequential images of a sample using an optical signal detection device; and determining a 
correction for a misalignment between each of a plurality of pairs of the sequential images, 

15 where at least one of the misalignments is due at least in part to a movement of the sample. The 
correction may then be used to compensate for each of the misalignments by automatically 
adjusting the optical signal detection device. 

[0025] The obtaining step and the determining step may be performed alternately or 
concurrently, for example. One embodiment comprises determining a correction for a 
20 misalignment between a pair of the sequential images less than about 2 seconds after obtaining 
the latter of the pair of the sequential images. In another embodiment, this takes less than about 
one second. 

[0026] In another aspect, the invention is directed to a method of determining a correction for a 
misalignment that includes validating the correction. Methods comprise the steps of: obtaining 
25 a plurality of sequential images of a sample using an optical signal detection device; determining 
a correction for a misalignment between at least two of the sequential images; and validating the 
correction for misalignment between two of the images. An embodiment further comprises 
compensating for the misalignment by automatically adjusting the optical signal detection device 
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5 according to the correction determined. In one embodiment, determining a misalignment 

correction between two images and validating the correction is performed in less than about one 
second. 

[0027] Methods of the invention comprise compensating for a misalignment by determining a 
correction for a misalignment between a pair of images, validating the misalignment, and 

10 automatically realigning one of the pair of images. The realignment may be performed during 
the acquisition of the images, or afterwards. 

Brief Description of the Drawings 
[0028] The objects and features of the invention can be better understood with reference to the 
drawings described below, and the claims. The drawings are not necessarily to scale, emphasis 

15 instead generally being placed upon illustrating the principles of the invention. In the drawings, 
like numerals are used to indicate like parts throughout the various views. 
[0029] Figure 1 A represents a 480 x 500 pixel image from a sequence of images of in vivo 
human cervix tissue and shows a 256 x 256 pixel portion of the image from which data is used in 
determining a correction for a misalignment between two images from a sequence of images of 

20 the tissue according to an illustrative embodiment of the invention. 

[0030] Figure IB depicts the image represented in Figure 1 A and shows a 128 x 128 pixel 
portion of the image, made up of 16 individual 32 x 32 pixel validation cells, from which data is 
used in performing a validation of the misalignment correction determination according to an 
illustrative embodiment of the invention. 

25 [0031] Figure 2A is a schematic flow diagram depicting steps in a method of determining a 
correction for a misalignment between two images due to at least in part to the movement of a 
sample according to an illustrative embodiment of the invention. 
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5 [0032] Figure 2B is a schematic flow diagram depicting steps in a version of the method 
shown in Figure 2A of determining a correction for a misalignment between two images due to 
at least in part to the movement of a sample according to an illustrative embodiment of the 
invention. 

[0033] Figure 2C is a schematic flow diagram depicting steps in a version of the method 
10 shown in Figure 2 A of determining a correction for a misalignment between two images due to 
at least in part to the movement of a sample according to an illustrative embodiment of the 
. invention. 

[0034] Figure 3 depicts a subset of adjusted images from a sequence of images of a tissue with 
an overlay of gridlines showing the validation cells used in validating the determinations of 
15 misalignment correction between the images according to an illustrative embodiment of the 
invention. 

[0035] Figure 4A depicts a sample image after application of a 9-pixel size (9 x 9) Laplacian 
of Gaussian filter (LoG 9 filter) on an exemplary image from a sequence of images of tissue 
according to an illustrative embodiment of the invention. 

20 [0036] Figure 4B depicts the application of both a feathering technique and a Laplacian of 
Gaussian filter on the exemplary unfiltered image used in Figure 4A to account for border 
processing effects according to an illustrative embodiment of the invention. 
[0037] Figure 5 A depicts a sample image after application of a LoG 9 filter on an exemplary 
image from a sequence of images of tissue according to an illustrative embodiment of the 

25 invention. 

[0038] Figure 5B depicts the application of both a Hamming window technique and a LoG 9 
filter on the exemplary unfiltered image used in Figure 5A to account for border processing 
effects according to an illustrative embodiment of the invention. 
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5 [0039] Figure 6 depicts the determination of a correction for misalignment between two 
images using methods including the application of LoG filters of various sizes, as well as the 
application of a Hamming window technique and a feathering technique according to illustrative 
embodiments of the invention. 

Description of the Illustrative Embodiment 
10 [0040] In general, the invention provides methods of determining a correction for a 

misalignment between images in a sequence due to movement of a sample. These methods are 
useful, for example, in the preparation of a sequence of images for analysis, as in medical 
diagnostics. 

[0041] In some diagnostic procedures, methods of the invention comprise applying an agent to 
15 a tissue in order to change its optical properties in a way that is indicative of the physiological 
state of the tissue. The rate and manner in which the tissue changes are important in the 
characterization of the tissue. 

[0042] Certain embodiments of the invention comprise automated and semi-automated 
analysis of diagnostic procedures that have traditionally required analysis by trained medical 

20 personnel. Diagnostic procedures which use automatic image-based tissue analysis provide 
results having increased sensitivity and/or specificity. See, e.g., co-owned U.S. patent 
application Serial No. 10/099,881, filed March 15, 2002, and co-owned U.S. patent application 
entitled, "Method and Apparatus for Identifying Spectral Artifacts," identified by Attorney 
Docket Number MDS-033, filed September 13, 2002, both of which are incorporated herein by 

25 reference. 

[0043] In order to facilitate such automatic analysis, it is often necessary to adjust for 
misalignments caused by sample movement that occurs during the diagnostic procedure. For 
example, during a given procedure, in vivo tissue may spatially shift within the image frame field 
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5 from one image to the next due to movement of the patient. Accurate diagnosis requires that this 
movement be taken into account in the automated analysis of the tissue sample. In some 
exemplary embodiments, spatial shift correction made at the time images are obtained is more 
accurate than correction made after all the images are obtained, since "on-the-fly" corrections 
compensate for smaller shifts occurring over shorter periods of time, rather than larger, more 

10 cumulative shifts occurring over longer periods of time. 

[0044] If a sample moves while a sequence of images is obtained, the procedure may have to 
be repeated. For example, this may be because the shift between consecutive images is too large 
to be accurately compensated for, or because a region of interest moves outside of a usable 
portion of the frame captured by the optical signal detection device. It is often preferable to 

15 compensate for misalignments resulting from sample movement during the collection of images 
rather than wait until the entire sequence of images has been obtained before compensating for 
misalignments. Stepwise adjustment of an optical signal detection device throughout image 
capture reduces the cumulative effect of sample movement. If adjustment is made only after an 
entire sequence is obtained, it may not be possible to accurately compensate for some types of 

20 sample movement. On-the-fly, stepwise compensation for misalignment reduces the need for 
retakes. 

[0045] On-the-fly compensation may also obviate the need to obtain an entire sequence of 
images before making the decision to abort a failed procedure, particularly when coupled with 
on-the-fly, stepwise validation of the misalignment correction determination. For example, if the 
25 validation procedure detects that a misalignment correction determination is either too large for 
adequate compensation to be made or is invalid, the procedure may be aborted before obtaining 
the entire sequence of images. It can be immediately determined whether or not the obtained 
data is useable. Retakes may be performed during the same patient visit; no follow-up visit to 
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5 repeat an erroneous test is required. A diagnostic test invalidated by excessive movement of the 
patient may be aborted before obtaining the entire sequence of images. 

[0046] In preferred embodiments, a determination of misalignment correction is expressed as a 
translational displacement in two dimensions, x and y. Here, x and y represent Cartesian 
coordinates indicating displacement on the image frame field plane. In other embodiments, 

10 corrections for misalignment are expressed in terms of non-Cartesian coordinate systems, such as 
biradical, spherical, and cylindrical coordinate systems, among others. Alternatives to Cartesian- 
coordinate systems may be useful, for example, where the image frame field is non-planar. 
[0047] Some types of sample motion - including rotational motion, warping, and local 
deformation - may result in an invalid misalignment correction determination, since it may be 

15 impossible to express certain instances of these types of sample motion in terms of a translational 
displacement, for example, in the two Cartesian coordinates x and y. It is noted, however, that in 
some embodiments, rotational motion, warping, local deformation, and/or other kinds of non- 
translational motion are acceptably accounted for by a correction expressed in terms of a 
translational displacement. The changing features of the tissue, as in acetowhitening, may also 

20 affect the determination of a misalignment correction. Image recording problems such as focus 
adjustment, missing images, blurred or distorted images, low signal-to-noise ratio (i.e. caused by 
glare), and computational artifacts may affect the correction determination as well. Therefore, 
validation of a determined correction is often required. In some embodiments, a validation step 
includes determining whether an individual correction for misalignment is erroneous, as well as 

25 determining whether to abort or continue the test in progress. Generally, validation comprises 
splitting at least a portion of each of a pair of images into smaller, corresponding units 
(subimages), determining for each of these smaller units a measure of the displacement that 



WO 2004/032058 PCT/US2003/030711 

-14- 

5 occurs within the unit between the two images, and comparing the unit displacements to the 
overall displacement between the two images. 

[0048] In certain embodiments, the method of validation takes into account the fact that 
features of a tissue sample may change during the capture of a sequence of images. For 
example, the optical intensity of certain regions of tissue change during an acetowhitening test. 

10 Therefore, in preferred embodiments, validation of a misalignment correction determination is 
performed using a pair of consecutive images. In this way, the difference between the 
corresponding validation cells of the two consecutive images is less affected by gradual tissue 
whitening changes, as compared with images obtained further apart in time. In some 
embodiments, validation is performed using pairs of nonconsecutive images taken within a 

15 relatively short period of time, compared with the time in which the overall sequence of images 
is obtained. In other embodiments, validation comprises the use of any two images in the 
sequence of images. 

[0049] In some exemplary embodiments, a determination of misalignment correction between 
two images may be inadequate if significant portions of the images are featureless or have low 

20 signal-to-noise ratio (i.e. are affected by glare). Similarly, validation using cells containing 
significant portions which are featureless or which have low signal-to-noise ratio may result in 
the erroneous invalidation of valid misalignment correction determinations in cases where the 
featureless portion of the overall image is small enough so that it does not adversely affect the 
misalignment correction determination. For example, analysis of featureless validation cells may 

25 produce meaningless correlation coefficients. One embodiment comprises identifying one or 
more featureless cells and eliminating them from consideration in the validation of a 
misalignment correction determination, thereby preventing rejection of a good misalignment 
correction. 
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5 [0050] A determination of misalignment correction may be erroneous due to a computational 
artifact of data filtering at the image borders. For example, in one exemplary embodiment, an 
image with large intensity differences between the upper and lower borders and/or the left and 
right borders of the image frame field undergoes Laplacian of Gaussian frequency domain 
filtering. Since Laplacian of Gaussian frequency domain filtering corresponds to cyclic 

10 convolution in the space-time domain, these intensity differences (discontinuities) yield a large 
gradient value at the image border, and cause the overall misalignment correction determination 
to be erroneous, since changes between the two images due to spatial shift are dwarfed by the 
edge effects. Certain embodiments employ pre-multiplication of image data by a Hamming 
window to remove or reduce this "wraparound error." Preferred embodiments employ image- 

15 blending techniques such as feathering, to smooth any border discontinuity, while requiring only 
a minimal amount of additional processing time. 

[0051] Figure 1 A represents a 480 x 500 pixel image 102 from a sequence of images of in vivo 
human cervix tissue and shows a 256 x 256 pixel portion 104 of the image from which data is 
used in identifying a misalignment correction between two images from a sequence of images of 

20 the tissue, according to an illustrative embodiment of the invention. Preferred embodiments 
comprise illuminating the tissue using either or both a white light source and a UV light source. 
The image 102 of Figure 1 A has a pixel resolution of about 0.054-mm. The embodiments 
described herein show images with pixel resolutions of about 0.0547-mm to about 0.0537-mm. 
Other embodiments have pixel resolutions outside this range. In some embodiments, the images 

25 of a sequence have an average pixel resolution of between about 0.044-mm and about 0.064-mm. 
In the embodiment of Figure 1 A, the central 256 x 256 pixels 104 of the image 102 are chosen 
for use in motion tracking. Other embodiments use regions of different sizes for motion 
tracking, and these regions are not necessarily located in the center of the image frame field. In 



WO 2004/032058 



-16- 



PCT/US2003/030711 



5 the embodiment of Figure 1 A, the method of motion tracking determines an x-displacement and 
a y-displacement corresponding to the translational shift (misalignment) between the 256 x 256 
central portions 104 of two images in the sequence of images. 

[0052] The determination of misalignment correction may be erroneous for any number of 
various reasons, including but not limited to non-translational sample motion (i.e. rotational 

10 motion, local deformation, and/or warping), changing features of a sample (i.e. whitening of 
tissue), and image recording problems such as focus adjustment, missing images, blurred or 
distorted images, low signal-to-noise ratio, and computational artifacts. Therefore, in preferred 
embodiments, validation comprises splitting an image into smaller units (called cells), 
determining displacements of these cells, and comparing the cell displacements to the overall 

15 displacement. Figure IB depicts the image represented in Figure 1A and shows a 128 x 128 
pixel portion 154 of the image, made up of 16 individual 32 x 32 pixel validation cells 156, from 
which data is used in performing a validation of the misalignment correction, according to an 
illustrative embodiment of the invention. 

[0053] Figure 2A, Figure 2B, and Figure 2C depict steps in illustrative embodiment methods 
20 of determining a misalignment correction between two images of a sequence, and methods of 
validating that determination. Steps 202 and 204 of Figure 2 A depict steps of developing data 
from an initial image with which data from a subsequent image are compared in order to 
determine a misalignment correction between the subsequent image and the initial image. An 
initial image "o" is preprocessed 202, then filtered 204 to obtain a matrix of values, for example, 
25 optical intensities, representing a portion of the initial image. In one embodiment, preprocessing 
includes transforming the three RGB color components into a single intensity component. An 
exemplary intensity component is CCIR 601, shown in Equation 1: 

I = 0.299R + 0.587G + 0.1 14B (1) 
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where I is the CCIR 601 "gray scale" intensity component, expressed in terms of red (R), green 
(G), and blue (B) intensities. CCIR 601 intensity may be used, for example, as a measure of the 
"whiteness" of a particular pixel in an image from an acetowhitening test. Different expressions 
for intensity may be used, and the choice may be geared to the specific type of diagnostic test 
conducted. In an alternative embodiment, a measure of radiant power as determined by a 
spectrometer may be used in place of the intensity component of Equation (1). Some 
embodiments comprise obtaining multiple types of optical signals simultaneously or 
contemporaneously; for example, some embodiments comprise obtaining a combination of two 
or more of the following signals: fluorescence spectra, reflectance (backscatter) spectra, and a 
video signal. Step 202 of Figure 2A is illustrated in blocks 240, 242, and 244 of Figure 2B, 
where block 240 represents the initial color image, "o", in the sequence, block 242 represents 
conversion of color data to gray scale using Equation 1, and block 244 represents the image of 
block 240 after conversion to gray scale. 

[0054] Step 204 of Figure 2A represents filtering a 256 x 256 portion of the initial image, for 
example, a portion analogous to the 256 x 256 central portion 104 of the image 102 of Figure 
1 A, using Laplacian of Gaussian filtering. Other filtering techniques are used in other 
embodiments. Preferred embodiments employ Laplacian of Gaussian filtering, which combines 
the Laplacian second derivative approximation with the Gaussian smoothing filter to reduce the 
high frequency noise components prior to differentiation. This filtering step may be performed 
by discrete convolution in the space domain, or by frequency domain filtering. The Laplacian of 
Gaussian (LoG) filter may be expressed in terms of x and y coordinates (centered on zero) as 
shown in Equation (2): 




(2) 
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5 where x and y are space coordinates and o is the Gaussian standard deviation. In one preferred 
embodiment, an approximation to the LoG function is used. In the embodiments described 
herein, approximation kernels of size 9x9, 21x21, and 31x31 are used. The Gaussian 
standard deviation a is chosen in certain preferred embodiments as shown in Equation (3): 

<j = LoG filter size / 8.49 (3) 

10 where LoG filter size corresponds to the size of the discrete kernel approximation to the LoG 
function (i.e. 9, 21, and 31 for the approximation kernels used herein). Other embodiments 
employ different kernel approximations and/or different values of Gaussian standard deviation. 
[0055] The LoG filter size may be chosen so that invalid scans are failed and valid scans are 
passed with a minimum of error. Generally, use of a larger filter size is better at reducing large 

15 structured noise and is more sensitive to larger image features and larger motion, while use of a 
smaller filter size is more sensitive to smaller features and smaller motion. One embodiment of 
the invention comprises using more than one filter size, adjusting to coordinate with the kind of 
motion being tracked and the features being imaged. 

[0056] Step 204 of Figure 2A is illustrated in Figure 2B in blocks 244, 246, and 248, where 
20 block 244 represents data from the initial image in the sequence after conversion to gray scale 
intensity, block 246 represents the application of the LoG filter, and block 248 represents the 256 
x 256 matrix of data values, G 0 (x,y), which is the "gold standard" by which other images are 
compared in validating misalignment correction determinations in this embodiment. As detailed 
in Figure 2C, preferred embodiments validate a misalignment correction determination by 
25 comparing a given image to its preceding image in the sequence, not by comparing a given 
image to the initial image in the sequence as shown in Figure 2B. Although Figure 2A, Figure 
2B, and Figure 2C show application of the LoG filter as a discrete convolution in the space 
domain, resulting in a standard expressed in space coordinates, other preferred embodiments 
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5 comprise applying the LoG filter in the frequency domain. In either case, the LoG filter is 
preferably zero padded to the image size. 

[0057] Steps 206 and 208 of Figure 2A represent preprocessing an image "i", for example, by 
converting RGB values to gray scale intensity as discussed above, and performing LoG filtering 
to obtain Gi(x,y), a matrix of values from image "i" which is compared with that of another 

10 image in the sequence in order to determine a misalignment correction between the two images. 
Steps 206 and 208 of Figure 2A are illustrated in Figure 2B in blocks 250, 252, 254, 256, and 
258, where fi(x,y) in block 250 is the raw image data from image "i" block 252 represents 
conversion of the fi(x,y) data to gray scale intensities as shown in block 254, and block 256 
represents application of the LoG filter on the data of block 254 to produce the data of block 258, 

15 Gi(x,y). 

[0058] Similarly, steps 212 and 214 of Figure 2A represent preprocessing an image "j", for 
example, by converting RGB values to gray scale intensity as discussed above, and performing 
LoG filtering to obtain Gj(x,y), a matrix of values from image "j" which is compared with image 
"i" in order to determine a measure of misalignment between the two images. In some preferred 

20 embodiments, image "j" is subsequent to image "i" in the sequence. In some preferred 

embodiments, "i" and "j" are consecutive images. Steps 212 and 214 of Figure 2A are illustrated 
in Figure 2B in blocks 264, 266, 268, 270, and 272, where "j" is "i+1", the image consecutive to 
image "i" in the sequence. In Figure 2B, block 264 is the raw "i+1" image data, block 266 
represents conversion of the "i+1" data to gray scale intensities as shown in block 268, and block 

25 270 represents application of the LoG filter on the data of block 268 to produce the data of block 
272, G i+ i(x,y). 

[0059] Steps 2 1 0 and 2 1 6 of Figure 2 A represent applying a Fourier transform, for example, a 
Fast Fourier Transform (FFT), using Gj(x,y) and Gj(x,y), respectively, to obtain Fi(u,v) and 
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5 Fj(u,v), which are matrices of values in the frequency domain corresponding to data from images 
"i" and "j" respectively. Steps 210 and 216 of Figure 2A are illustrated in Figure 2B by blocks 
258, 260, 262, 272, 274, and 276, where "j" is "i+1", the image consecutive to image "i" in the 
sequence. In Figure 2B, block 258 represents the LoG filtered data, Gi(x,y), corresponding to 
image "i", and block 260 represents taking the Fast Fourier Transform of Gi(x,y) to obtain 

10 Fj(u,v), shown in block 262. Similarly, in Figure 2B block 272 is the LoG filtered data, 
Gi+i(x,y), corresponding to image "i+1", and block 274 represents taking the Fast Fourier 
Transform of Gi+i(x,y) to obtain Fj+i(u,v), shown in block 276. 
[0060] Step 2 1 8 of Figure 2 A represents computing the cross correlation 
Fj(u,v) F*(u,v), where Fj(u,v) is the Fourier transform of data from image "i", F*j(u,v) is the 

15 complex conjugate of the Fourier transform of data from image "j", and u and v are frequency 
domain variables. The cross-correlation of two signals of length Ni and N2 provides N1+N2-I 
values; therefore, to avoid aliasing problems due to under-sampling, the two signals should be 
padded with zeros up to N1+N2-I samples. Step 218 of Figure 2A is represented in Figure 2B by 
blocks 262, 276, and 278. Block 278 of Figure 2B represents computing the cross correlation, 

20 Fj(u,v) F*i+i(u,v), using Fi(u,v), the Fourier transform of data from image "i", and F*i+i(u,v), the 
complex conjugate of the Fourier transform of data from image "i+1". The cross-correlation 
may also be expressed as c(kj) in Equation (4): 

cikfi-Whip&hip-hq-r) (4) 
where variables (k,l) can be thought of as the shifts in each of the x- and y-directions which are 

25 being tested in a variety of combinations to determine the best measure of misalignment between 
two images I\ and and where p and q are matrix element markers. 

[0061] Step 220 of Figure 2 A represents computing the inverse Fourier transform of the cross- 
correlation computed in step 218. Step 220 of Figure 2 A is represented in Figure 2B by block 
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5 280. The resulting inverse Fourier transform maps how well the 256 x 256 portions of images 
"i" and "j" match up with each other given various combinations of x- and y-shifts. Generally, 
the normalized correlation coefficient closest to 1.0 corresponds to the x-shift and y-shift 
position providing the best match, and is determined from the resulting inverse Fourier 
transform. In a preferred embodiment, correlation coefficients are normalized by dividing 
10 matrix values by a scalar computed as the product of the square root of the (0,0) value of the 
awto-correlation of each image. In this way, variations in overall brightness between the two 
images have a more limited effect on the correlation coefficient, so that the actual movement 
within the image frame field between the two images is better reflected in the misalignment 
determination. 

15 [0062] Step 222 of Figure 2A represents determining misalignment values d x , d y , d, sum(dx), 
sum(dy), and Sum(dj), where d x is the computed displacement between the two images "i" and 
"j" in the x-direction, d y is the computed displacement between the two images in the y- 
direction, d is the square root of the sum d x 2 +d y 2 and represents an overall displacement between 
the two images, sum(d x ) is the cumulative x-displacement between the current image "j" and the 

20 first image in the sequence "o", sum(d y ) is the cumulative y-displacement between the current 
image "j" and the first image in the sequence "o", and Sum(dj) is the cumulative displacement, d, 
between the current image "j" and the first image in the sequence "o". Step 222 of Figure 2A is 
represented in Figure 2B by blocks 282, 284, and 286. Blocks 284 and 286 represent finding the 
maximum value in the data of block 282 in order to calculate d x , d y , d, sum(d x ), sum(d y ), and 

25 Sum(di+i) as described above, where image "j" in Figure 2A is "i+1 " in Figure 2B, the image 
consecutive to image "i". 

[0063] Steps 224, 226, and 228 of Figure 2A represent one method of validating the 
misalignment correction determined for image "j" in step 222 of Figure 2A. This method of 
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validating misalignment correction is represented in blocks 287, 289, 291, 296, 297, and 298 of 
Figure 2C. Another method of validating a misalignment correction is represented in steps 230, 
232, and 234 of Figure 2A; and this method is represented in blocks 288, 290, 292, 293, 294, and 
295 of Figure 2B. Figure 2C is a schematic flow diagram depicting steps in a version of the 
methods shown in Figure 2 A of determining a correction for a misalignment between two 
images in which validation is performed using data from two consecutive images. Preferred 
embodiments comprise using consecutive or near-consecutive images to validate a misalignment 
correction determination, as in Figure 2C. Other embodiments comprise using the initial image 
to validate a misalignment correction determination for a given image, as in Figure 2B. 
[0064J In Figure 2A, step 224 represents realigning Gj(x,y), the LoG-filtered data from image 
"j", to match up with Gi(x,y), the LoG-filtered data from image "i" using the misalignment 
values d x and d y determined in step 222. In preferred embodiments, image "j" is consecutive to 
image "i" in the sequence of images. Here, image "j" is image "i+1" such that Gi(x,y) is aligned 
with G i+ i(x,y) as shown in block 287 of Figure 2C. Similarly, in Figure 2A, step 230 represents 
realigning Gj(x,y), the LoG-filtered data from image "j", to match up with G 0 (x,y), the LoG- 
filtered "gold standard" data from the initial image "o", using the displacement values sum(d x ) 
and sum(d y ) determined in step 222. Step 230 of Figure 2A is represented in block 288 of Figure 
2B. 

[0065] Step 226 of Figure 2 A represents comparing corresponding validation cells from 
Gj(x,y) and Gj(x,y) by computing correlation coefficients for each cell. This is represented 
schematically in Figure 2C by blocks 289, 291, 296, 297, and 298 for the case where j = i+1 . 
First, a 128 x 128 pixel central portion of the realigned G i+ i(x,y) is selected, and the 
corresponding 128 x 128 pixel central portion of Gi(x,y) is selected, as shown in blocks 289 and 
291 of Figure 2C. An exemplary 128 x 128 pixel validation region 154 is shown in Figure IB. 



WO 2004/032058 



PCT/US2003/030711 



5 Then, the embodiment comprises computing a correlation coefficient for each of 1 6 validation 
cells. An exemplary validation cell from each of the realigned Gi+i(x,y) matrix 291 and Gj(x,y) 
matrix 289 is shown in blocks 297 and 296 of Figure 2C. The validation cells are as depicted in 
the 32 x 32 pixel divisions 156 of the 128 x 128 pixel validation region 154 of Figure IB. 
Different embodiments use different numbers and/or different sizes of validation cells. 

10 Correlation coefficients are computed for each of the 16 cells, as shown in block 298 of Figure 
2C. Each correlation coefficient is a normalized cross-correlation coefficient as shown in 
Equation (5): 



where c'(m,n) is the normalized cross-correlation coefficient for the validation cell (m,n), m is an 
15 integer 1 to 4 corresponding to the column of the validation cell whose correlation coefficient is 
being calculated, n is an integer 1 to 4 corresponding to the row of the validation cell whose 
correlation coefficient is being calculated, p and q are matrix element markers, Ii[p,q] are 
elements of the cell in column m and row n of the 128 x 128 portion of the realigned image 
shown in block 291 of Figure 2C, and h\p,q] are elements of the cell in column m and row n of 
20 the 128 x 128 portion of Gi(x,y) shown in block 289 of Figure 2C. Here, p « 1 to 32 and q = 1 to 
32, and the sums shown in Equation (5) are performed over p and q. The cross-correlation 
coefficient of Equation (5) is similar to an auto-correlation in the sense that a subsequent image 
is realigned with a prior image based on the determined misalignment correction so that, ideally, 
the aligned images appear to be identical. A low value of c'(m,n) indicates a mismatching 
25 between two corresponding cells. The misalignment correction determination is then either 
validated or rejected based on the values of the 16 correlation coefficients computed in step 298 
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5 of Figure 2C. For example, each correlation coefficient may be compared against a threshold 
maximum value. This corresponds to step 228 of Figure 2 A. 

[0066] Step 232 of Figure 2 A represents comparing corresponding validation cells from 
Gj(x,y) and G 0 (x,y) by computing correlation coefficients for each cell. This is represented 
schematically in Figure 2B by blocks 290, 292, 293, 294, and 295 for the case where j = i+1. 

10 First, a 128 x 128 pixel central portion of the realigned Gi+i(x,y) is selected, and the 

corresponding 128 x 128 pixel central portion of G 0 (x,y) is selected, as shown in blocks 292 and 
290 of Figure 2B. An exemplary 128 x 128 pixel validation region 154 is shown in Figure IB. 
Then, the embodiment comprises computing a correlation coefficient for each of the 16 
validation cells. An exemplary validation cell from each of the realigned Gj+i(x,y) matrix 292 

15 and G 0 (x,y) matrix 290 is shown in blocks 294 and 293 of Figure 2B. The validation cells are as 
depicted in the 32 x 32 pixel divisions 156 of the 128 x 128 pixel validation region 154 of Figure 
IB. Different embodiments use different numbers of and/or different sizes of validation cells. 
Correlation coefficients are computed for each of the 16 cells, as shown in block 295 of Figure 
2B. Each correlation coefficient is a normalized "autocorrelation coefficient as shown in 

20 Equation (5) above, where Ii[p,q] are elements of the cell in column m and row n of the 128 x 
128 portion of the realigned subsequent image shown in block 292 of Figure 2B, and l2[p>q] are 
elements of the cell in column m and row n of the 128 x 128 portion of G 0 (x,y) shown in block 
290 of Figure 2B. A low value of c'(m,n) indicates a mismatching between two corresponding 
cells. The misalignment determination is then either validated or rejected based on the values of 

25 the 16 correlation coefficients computed in step 295 of Figure 2C. This corresponds to step 234 
of Figure 2 A. 

[0067] In an illustrative embodiment, determinations of misalignment correction and 
validation of these determinations as shown in each of Figure 2A, Figure 2B, and Figure 2C are 
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5 performed using a plurality of the images in a given sequence. In preferred embodiments, 
determinations of misalignment correction and validations thereof are performed while images 
are being obtained, so that an examination in which a given sequence of images is obtained may 
be aborted before all the images are obtained. In some embodiments, a misalignment correction 
is determined, validated, and compensated for by adjusting the optical signal detection device 

10 obtaining the images. In certain embodiments, an adjustment of the optical signal detection 
device is made after each of a plurality of images are obtained. In certain embodiments, an 
adjustment, if required by the misalignment correction determination, is made after every image 
subsequent to the first image (except the last image), and prior to the next consecutive image. In 
one embodiment, a cervical tissue scan comprising a sequence of 13 images is performed using 

15 on-the-fly misalignment correction determination, validation, and camera adjustment, such that 
the scan is completed in about 12 seconds. Other embodiments comprise obtaining sequences of 
any number of images in more or less time than indicated here. 

[0068] Each of steps 228 and 234 of the embodiment of Figure 2 A represents applying a 
validation algorithm to determine at least the following: (1) whether the misalignment correction 

20 can be made, for example, by adjusting the optical signal detection device, and (2) whether the 
misalignment correction determined is valid. In an exemplary embodiment, the validation 
algorithm determines that a misalignment correction cannot be executed during an 
acetowhitening exam conducted on cervical tissue in time to provide sufficiently aligned 
subsequent images, if either of conditions (a) or (b) is met, as follows: (a) du the displacement 

25 between the current image "i" and the immediately preceding image is greater than 0.55- 
mm or (b) Sum(dj), the total displacement between the current image and the first image in the 
sequence, "o", is greater than 2.5-mm. If either of these conditions is met, the exam in progress 
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5 is aborted, and another exam must be performed. Other embodiments may comprise the use of 
different validation rules. 

[0069] In the exemplary embodiment above, validation is performed for each determination of 
misalignment correction by counting how many of the correlation coefficients c'r(m,n) shown in 
Equation (5), corresponding to the 16 validation cells, is less than 0.5. If this number is greater 

10 than 1, the exam in progress is aborted. Other embodiments may comprise the use of different 
validation rules. Gradual changes in image features, such as acetowhitening of tissue or changes 
in glare, cause discrepancies which are reflected in the correlation coefficients of the validation 
cells, but which do not represent a spatial shift. Thus, in preferred embodiments, the validation 
is performed as shown in Figure 2C, where validation cells of consecutive images are used to 

15 calculate the correlation coefficients. In other embodiments, the validation is performed as 
shown in Figure 2B, where validation cells of a current image, "i" and an initial image of the 
sequence, "o", are used to calculate the correlation coefficients of Equation (5). 
[0070] Figure 3 depicts a subset of adjusted, filtered images 302, 306, 3 1 0, 3 14, 3 1 8, 322 from 
a sequence of images of a tissue with an overlay of gridlines showing the validation cells used in 

20 validating the determinations of misalignment correction between the images, according to an 
illustrative embodiment of the invention. By performing validation according to Figure 2C, 
using consecutive images to calculate the correlation coefficients of Equation (5), the number of 
validation cells with correlation coefficient below 0.5 for the misalignment-corrected images of 
Figure 3 is 0, 1, 0, 0, and 1 for images 306, 310, 314, 318, and 322, respectively. Since none of 

25 the images have more than one coefficient below 0.5, this sequence is successful and is not 
aborted. This is a good result in the example of Figure 3, since there is no significant tissue 
movement occurring between the misalignment-corrected images. There is only a gradually 
changing glare, seen to move within the validation region 304, 308, 312, 316, 320, 324 of each 



WO 2004/032058 



-27- 



PCT/US2003/030711 



5 image. In an embodiment in which validation is performed as in Figure 2B, the number of 
validation cells with correlation coefficient below 0.5 for the misalignment-corrected images of 
Figure 3 is 3, 4, 5, 5, and 6 for images 306, 310, 314, 318, and 322, respectively. This is not a 
good result in this example, since the exam would be erroneously aborted, due only to gradual 
changes in glare or whitening of tissue, not uncompensated movement of the tissue sample. 
1 0 [0071] In a preferred embodiment, validation cells that are featureless or have low signal-to- 
noise ratio are eliminated from consideration. These cells can produce meaningless correlation 
coefficients. Featureless cells in a preferred embodiment are identified and eliminated from 
consideration by examining the deviation of the sum squared gradient of a given validation cell 
from the mean of the sum squared gradient of all cells as shown in the following exemplary rule: 

15 Rule : If ssgi(m,n) < Mean[ssg(m,n)] - STD[ssg(m,n)], then set c' i(m,n) = 1 .0. 

where c'i(m,n) is the correlation of the given validation cell "1", ssgi(m,n) = £ E Ii 2 [p,q], m = 1 
to 4, n = 1 to 4, Ii[p,q] is the matrix of values of the given validation cell "1", p = 1 to 32, q = 1 
to 32, the summations EE are performed over pixel markers p and q, Mean[ssg(m,n)] is the 
mean of the sum squared gradient of all 16 validation cells, and STD[ssg(m,n)] is the standard 

20 deviation of the sum squared gradient of the given validation cell "1" from the mean sum 

squared gradient. By setting c'i(m,n) = 1.0 for the given validation cell, the cell does not count 
against validation of the misalignment correction determination in the rubrics of either step 228 
or step 234 of Figure 2A, since a correlation coefficient of 1.0 represents a perfect match. 
[0072] If an image has large intensity differences between the upper and lower borders and/or 

25 the left and right borders of the image frame field, LoG filtering may result in "wraparound 
error." A preferred embodiment employs an image blending technique such as "feathering" to 
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5 smooth border discontinuities, while requiring only a minimal amount of additional processing 
time. 

[0073] Figure 4A depicts a sample image 402 after application of a 9-pixel size [9 x 9] 
Laplacian of Gaussian filter (LoG 9 filter) on an exemplary image from a sequence of images of 
tissue, according to an illustrative embodiment of the invention. The filtered intensity values are 

10 erroneous at the top edge 404, the bottom edge 406, the right edge 410, and the left edge 408 of 
the image 402. Since LoG frequency domain filtering corresponds to cyclic convolution in the 
space-time domain, intensity discontinuities between the top and bottom edges of an image and 
between the right and left edges of an image result in erroneous gradient approximations. These 
erroneous gradient approximations can be seen in the dark stripe on the right edge 410 and 

15 bottom edge 406 of the image 402, as well as the light stripe on the top edge 404 and the left 
edge 408 of the image 402. This often results in a misalignment correction determination that is 
too small, since changes between the images due to spatial shift are dwarfed by the edge effects. 
A preferred embodiment uses a "feathering" technique to smooth border discontinuities and 
reduce "wraparound error." 

20 [0074] Feathering comprises removal of border discontinuities prior to application of a filter. 
In preferred embodiments, feathering is performed on an image before LoG filtering, for 
example, between steps 206 and 208 in Figure 2A. In embodiments where LoG filtering is 
performed in the frequency domain (subsequent to Fourier transformation), feathering is 
preferably performed prior to both Fourier transformation and LoG filtering. For one- 

25 dimensional image intensity functions Ii(x) and h(x) that are discontinuous at x = Xo, an 
illustrative feathering algorithm is as follows: 

^) = / 1 (x)./(^. + 0.5) and I' 2 (x) = I 2 (x)> (l-/(^ + 0.5)), 
a d 
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0 jc<0 

3x 2 -2x 3 0< jc <1 , (6) 
0 x>l 



where I\Qt) and l2'(x) are the intensity functions Ii(x) and I 2 (x) after applying the feathering 
algorithm of Equation (6), and d is the feathering distance chosen. The feathering distance, d, 
adjusts the tradeoff between removing wraparound error and suppressing image content. 
[0075] Figure 4B depicts the application of both a feathering technique and a LoG filter on the 

10 same unfiltered image used in Figure 4 A. The feathering is performed to account for border 
processing effects, according to an illustrative embodiment of the invention. Here, a feathering 
distance, d, of 20 pixels was used. Other embodiments use other values of d. The filtered image 
420 of Figure 4B does not display uncharacteristically large or small gradient intensity values at 
the top edge 424, bottom edge 426, right edge 430, or left edge 428, since discontinuities are 

15 smoothed prior to LoG filtering. Also, there is minimal contrast suppression of image detail at 
the borders. Pixels outside the feathering distance, d, are not affected. The use of feathering 
here results in more accurate determinations of misalignment correction between two images in a 
sequence of images. 

[0076] Another method of border smoothing is multiplication of unfiltered image data by a 
20 Hamming window. In some embodiments, a Hamming window function is multiplied to image 
data before Fourier transformation so that the border pixels are gradually modified to remove 
discontinuities. However, application of the Hamming window suppresses image intensity as 
well as gradient information near the border of an image. 

[0077] Figure 5A is identical to Figure 4A and depicts the application of a LoG 9 filter on an 
25 exemplary image from a sequence of images of tissue according to an illustrative embodiment of 
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5 the invention. The filtered intensity values are erroneous at the top edge 404, the bottom edge 
406, the right edge 410, and the left edge 408 of the image 402. 

[0078] Figure 5B depicts the application of both a Hamming window and a LoG 9 filter on the 
same unfiltered image used in Figure 5 A. Hamming windowing is performed to account for 
border processing effects, according to an illustrative embodiment of the invention. Each of the 
10 edges 524, 526, 528, 530 of the image 520 of Figure 5B no longer show the extreme filtered 
intensity values seen at the edges 404, 406, 408, 410 of the image 402 of Figure 5 A. However, 
there is a greater suppression of image detail in Figure 5B than in Figure 4B. Thus, for this 
particular embodiment, application of the feathering technique is preferred over application of 
Hamming windowing. 

15 [0079] A skilled artisan knows other methods of smoothing border discontinuities. Another 
embodiment comprises removing cyclic convolution artifacts by zero padding the image prior to 
frequency domain filtering to assure image data at an edge would not affect filtering output at the 
opposite edge. This technique adds computational complexity and may increase processing 
time. 

20 [0080] Figure 6 depicts the determination of a misalignment correction between two images 
using methods including the application of LoG filters of various sizes, as well as the application 
of a Hamming window technique and a feathering technique, according to illustrative 
embodiments of the invention. Image 602 and image 604 at the top of Figure 6 are consecutive 
images from a sequence of images of cervix tissue obtained during a diagnostic exam, each with 

25 a pixel resolution of about 0.054-mm. Figure 6 depicts the application of four different image 
filtering algorithms: (1) Hamming window with LoG 9 filtering, (2) feathering with LoG 9 
filtering, (3) feathering with LoG 21 filtering, and (4) feathering with LoG 31 filtering. Each of 
these algorithms are implemented as part of a misalignment correction determination and 
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5 validation technique as illustrated in Figure 2A and Figure 2C, and values of d x and d y between 
images 602 and 604 of Figure 6 are determined using each of the four filtering algorithms. For 
image 602, each of the four different image filtering algorithms (1) - (4) listed above are applied, 
resulting in images 606, 610, 614, and 618, respectively, each having 256 x 256 pixels. The four 
different image filtering algorithms are also applied for image 604, resulting in images 608, 612, 

10 616, and 620, respectively, each having 256 x 256 pixels. Values of (d x , d y ) determined using 
Hamming + LoG 9 filtering are (-7, 0), expressed in pixels. Values of (d x , d y ) determined using 
feathering + LoG 9 filtering are (-2, -10). Values of (d x , d y ) determined using feathering + LoG 
21 filtering are (-1, -9). Values of (dx, d y ) determined using feathering 4- LoG 31 filtering are (0, 
-8). All of the displacement values determined using feathering are close in this embodiment, 

15 and agree well with visually-verified displacement. However, in this example, the displacement 
values determined using Hamming windowing are different from those obtained using the other 
three filtering methods, and result in a misalignment correction that does not agree well with 
visually-verified displacement. Thus, for this example, feathering works best since it does not 
suppress as much useful image data. 

20 [0081] The effect of the filtering algorithm employed, as well as the choice of validation rules 
are examined by applying combinations of the various filtering algorithms and validation rules to 
pairs of sequential images of tissue and determining the number of "true positives" and "false 
positives" identified. A true positive occurs when a bad misalignment correction determination 
is properly rejected by a given validation rule. A false positive occurs when a good 

25 misalignment correction determination is improperly rejected as a failure by a given validation 
rule. The classification of a validation result as a "true positive" or a "false positive" is made by 
visual inspection of the pair of sequential images. In preferred embodiments, whenever true 
failures occur, the scan should be aborted. Some examples of situations where true failures 
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5 occur in certain embodiments include image pairs between which there is one or more of the 
following: a large non-translational deformation such as warping or tilting; a large jump for 
which motion tracking cannot compute a correct translational displacement; rotation greater than 
about 3 degrees; situations in which a target laser is left on; video system failure such as blur, 
dark scan lines, or frame shifting; cases where the image is too dark and noisy, in shadow; cases 
10 where a vaginal speculum (or other obstruction) blocks about half the image; other obstructions 
such as sudden bleeding. 

[0082] In one embodiment, a set of validation rules is chosen such that true positives are 
maximized and false positives are minimized. Sensitivity and specificity can be adjusted by 
adjusting choice of filtering algorithms and/or choice of validation rules. Table 1 shows the 

15 number of true positives (true failures) and false positives (false failures) determined by a 
validation rule as depicted in Figure 2A and Figure 2C where validation is determined using 
consecutive images. Table 1 shows various combinations of filtering algorithms and validation 
rules. The four filtering algorithms used are (1) Hamming windowing with LoG 9 filtering, (2) 
feathering with LoG 9 filtering, (3) feathering with LoG 21 filtering, and (4) feathering with LoG 

20 3 1 filtering. The values, c'(m,n), correspond to the normalized "autocorrelation coefficient of 
Equation (5) whose value must be met or exceeded in order for a validation cell to "pass" in an 
embodiment. The '"Number Threshold" column indicates the maximum number of "failed" 
validation cells, out of the 1 6 total cells, that are allowed for a misalignment correction 
determination to be accepted in an embodiment. If more than this number of validation cells fail, 

25 then the misalignment correction determination is rejected. 
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5 Table 1: True positives and false positives of validation determinations for embodiments 
using various combinations of filtering algorithms and validation rules. 





c'(m,n) 


Number 
Threshold 


TP 


FP 


Hamming LoG 9 


-0.1 


1 


34 


28 


Feathering LoG 9 


-0.1 


3 


19 


17 


Feathering LoG 21 


0.3 


2 


46 


10 


0.35 


3 


52 


4 


Feathering LoG 31 


0.5 


3 


48 


3 



[0083] For the given set of cervical image pairs on which the methods shown in Table 1 were 
10 applied, feathering performs better than Hamming windowing, since there are more true 
positives and fewer false positives. Among different LoG filter sizes, LoG 21 and LoG 3 1 
performs better than LoG 9 for both tracking and validation here. The LoG 21 filter is more 
sensitive to rotation and deformation than the LoG 3 1 filter for these examples. Preferred 
embodiments for the determination and validation of misalignment corrections between 256 x 
15 256 pixel portions of images of cervical tissue with pixel resolution of about 0.054-mm employ 
one or more of the following: (1) use of feathering for image border processing, (2) application 
of LoG 21 filter, (3) elimination of validation cells with low signal-to-noise ratio, and (4) use of 
consecutive images for validation. 

Equivalents 

20 [0084] While the invention has been particularly shown and described with reference to 

specific preferred embodiments, it should be understood by those skilled in the art that various 
changes in form and detail may be made therein without departing from the spirit and scope of 
the invention as defined by the appended claims. 
[0085] What is claimed is: 
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11. A method of compensating for image misalignment, the method comprising the steps 

2 of: 

3 obtaining a sequence of images of a tissue sample; and 

4 correcting for misalignment between at least two of the images, said misalignment being 

5 due at least in part to movement of the tissue sample. 

1 2. The method of claim 1 , wherein the correcting step is performed in real time. 

1 3 . The method of claim 1 , wherein the correcting step comprises adjusting an optical signal 

2 detection device used to obtain the sequence of images. 

1 4. The method of claim 3, wherein the correcting step comprises adjusting a position of a 

2 component of the optical signal detection device. 

1 5. The method of claim 4, wherein the component comprises a mirror. 

1 6. The method of claim 1, wherein the tissue sample is an in-situ tissue sample and wherein 

2 the misalignment is due at least in part to patient motion. 

1 7. The method of claim 1 , further comprising the step of applying a contrast agent to the 

2 tissue sample. 

1 8. The method of claim 1 , wherein the correcting step comprises electronically adjusting at 

2 least one of the images. 

1 9. The method of claim 1 , wherein the at least two images are consecutive images. 

1 10. The method of claim 1 , wherein the correcting step comprises the step of filtering a 

2 subset of data from a first image of the sequence of images. 

1 11. The method of claim 10, wherein the correcting step comprises the step of preprocessing 

2 the subset of data prior to the filtering. 

1 12. The method of claim 10, wherein the filtering step comprises at least one of frequency 

2 domain filtering and discrete convolution in the space domain. 
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1 13. The method of claim 1 0, wherein the filtering step comprises Laplacian of Gaussian 

2 filtering. 

1 14. The method of claim 10, wherein the filtering step comprises using a feathering 

2 technique. 

1 15. The method of claim 1 0, wherein the filtering step comprises using a Hamming window. 

1 1 6. The method of claim 1 , wherein the correcting step comprises computing a cross 

2 correlation using data from two of the images. 

1 1 7. The method of claim 1 6, wherein the computing of the cross correlation comprises 

2 computing a product represented by 

3 Fi(u,v)F*j(u 5 v), 

4 where Fj(u,v) is a Fourier transform of data derived from a subset of data from a first image, i, of 

5 the sequence of images, F*j(u,v) is a complex conjugate of a Fourier transform of data derived 

6 from a subset of data from a second image, j, of the sequence of images, and u and v are 

7 frequency domain variables. 

1 18. The method of claim 1 7, wherein the computing of the cross correlation comprises 

2 computing an inverse Fourier transform of the product. 

1 19. The method of claim 1, wherein the tissue sample comprises cervical tissue. 

1 20. The method of claim 1 , wherein the tissue sample comprises at least one member of the 

2 group consisting of colorectal tissue, gastroesophageal tissue, urinary bladder tissue, lung tissue, 

3 and skin tissue. 

1 21 . The method of claim 1 , wherein the tissue sample comprises epithelial cells. 

1 22. The method of claim 1 , wherein the obtaining step comprises obtaining the sequence of 

2 images of the tissue sample during application of a chemical agent to the tissue sample. 
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1 23 . The method of claim 1 , wherein the obtaining step comprises obtaining the sequence of 

2 images of the tissue sample after application of a chemical agent to the tissue sample. 

1 24. The method of claim 23, wherein the chemical agent is selected from the group 

2 consisting of acetic acid, formic acid, propionic acid, and butyric acid. 

1 25. The method of claim 23, wherein the chemical agent is selected from the group 

2 consisting of Lugoi's iodine, Shiller's iodine, methylene blue, toluidine blue, indigo carmine, 

3 indocyanine green, and fluorescein. 

1 26. The method of claim 1 , wherein the obtaining step comprises obtaining the sequence of 

2 images of the tissue sample during an acetowhitening test. 

1 27. The method of claim 1 , wherein the movement of the tissue sample is relative to an 

2* optical signal detection device and comprises at least one member of the group consisting of 

3 translational motion, rotational motion, warping, and local deformation. 

1 28. The method of claim 1 , wherein one or more images of the sequence of images comprise 

2 measurements of an optical signal from the tissue sample. 

1 29. The method of claim 28, wherein the optical signal comprises visible light. 

1 30. The method of claim 28, wherein the optical signal comprises fluorescent light. 

1 31. The method of claim 28, wherein the optical signal is emitted by the tissue sample. 

1 32. The method of claim 28, wherein the optical signal is reflected by the tissue sample. 

1 33. The method of claim 28, wherein the optical signal is transmitted through the tissue 

2 sample. 

1 34. A method of validating a correction for an image misalignment, the method comprising 

2 the steps of: 

3 adjusting at least one of two or more images using a correction for an image 

4 misalignment between the two or more images; 
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5 defining one or more validation cells, each of which includes a common area of the two 

6 or more adjusted images; 

7 computing for each of the one or more validation cells a measure of displacement 

8 between the two or more adjusted images using data from the two or more adjusted images 

9 corresponding to each of the one or more validation cells; and 

10 validating the correction for the image misalignment by comparing at least one of the 

1 1 measures of displacement with a threshold value. 

1 35. A method of validating a correction for an image misalignment, the method comprising 

2 the steps of: 

3 defining one or more validation cells within a bounded image plane; 

4 computing for each of the one or more validation cells a measure of displacement 

5 between two or more images bound by the image plane using data from the two or more images 

6 corresponding to each of the one or more validation cells; 

7 validating a correction for an image misalignment between the two or more images by 

8 comparing at least one of the measures of displacement with the correction. 

1 36. The method of claim 34, wherein the images are images of an in-situ tissue sample, and 

2 wherein the image misalignment is due at least in part to patient motion. 

1 37. The method of claim 34, wherein the images are images of an in-situ tissue sample that 

2 has been treated with a contrast agent 

1 38. The method of claim 34, wherein the one or more validation cells comprise a subset of a 

2 bounded image plane common to the two or more images. 

1 39. The method of claim 34, wherein the two or more images are consecutive images. 
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2 40. The method of claim 38, wherein the one or more validation cells comprise a central 

3 portion of the bounded image plane. 

1 41 . The method of claim 38, wherein the bounded image plane has an area about four times 

2 larger than the total area of the one or more validation cells. 

1 42. The method of claim 34, wherein the validating step comprises eliminating from 

2 consideration one or more of the measures of displacement for one or more of the one or more 

3 validation cells. 

1 43. The method of claim 42, wherein the eliminating of the one or more measures of 

2 displacement comprises calculating a sum squared gradient for at least one of the one or more 

3 validation cells. 

1 44. A method of compensating for an image misalignment, the method comprising the steps 

2 of: 

3 obtaining a set of sequential images of a tissue sample; and 

4 correcting for a misalignment between each of a plurality of pairs of the sequential 

5 images, the misalignment due at least in part to movement of the tissue sample. 

1 45. The method of claim 44, wherein the tissue sample is an in-situ tissue sample and 

2 wherein the misalignment is due at least in part to patient motion. 

1 46. The method of claim 44, further comprising the step of applying to the sample a contrast 

2 agent. 

1 47. The method of claim 44, wherein the obtaining step and the correcting step are performed 

2 alternately. 

1 48. The method of claim 44, wherein the obtaining step and the correcting step are performed 

2 substantially concurrently. 
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1 49. The method of claim 44, wherein the correcting step comprises determining a correction 

2 for a misalignment between a pair of the sequential images less than about 2 seconds after the 

3 obtaining of the latter of the pair of the sequential images. 

1 50. The method of claim 44, wherein the correcting step comprises determining a correction 

2 for a misalignment between a pair of the sequential images less than about one second after the 

3 obtaining of the latter of the pair of the sequential images. 

1 51 . A method of validating a correction for an image misalignment, the method comprising 

2 the steps of: 

3 obtaining a plurality of sequential images of a sample using an optical signal detection 

4 device; 

5 determining a correction for a misalignment between at least two of the sequential 

6 images, the misalignment due at least in part to a movement of the sample; and 

7 validating the correction between at least a first image and a second image of the plurality 

8 of sequential images. 

1 52. The method of claim 51, wherein the sample is an in-situ tissue sample and wherein the 

2 misalignment is due at least in part to patient motion. 

1 53 . The method of claim 5 1 , further comprising the step of applying a contrast agent to the 

2 sample. 

1 54. The method of claim 5 1 , wherein the determination of a correction for a misalignment 

2 between a first and a second image and the validation of said correction are performed in less 

3 than about one second. 

1 55. The method of claim 5 1 , further comprising the step of: 

2 adjusting the optical signal detection device using the correction. 

1 56. A method of dynamically compensating for image misalignment, the method comprising 

2 the steps of: 
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obtaining a sequence of images of a tissue sample; and 
correcting in real time for misalignment between at least two of the images, the 
misalignment due at least in part to movement of the tissue sample. 
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(57) Abstract: The invention provides methods of 
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in a sequence of images of a sample. The methods may be 
applied in deciding whether a correction for misalignment 
accurately accounts for sample motion. 
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Via Certified Mail 

Return Receipt Requested 

Receipt No. 7004 2510 0001 6232 9125 

Ms. Anna Bogdanova 
ARS-Patent 

Suite 331,SHVEDSKY per 2 
St. Petersburg 
RU 191186 

RUSSIAN FEDERATION 

Re: PRI-MED Russian Trademark Registration 

Goodwin Procter Trademark File No. IRO-621 

Confidential & Privileged Attorney-Client Communication 

Dear Anna: 

I am pleased to forward to you the enclosed original Certificate of Registration, number 306400, for 
the mark SCOOBA, which was issued to the company on May 10, 2006 by the Russian Agency for 
Patents & Trademarks, for use in connection with the following goods and services (translated from 
the Russian): 



You should keep this original certificate in a secure place with the company's other important papers 
and records. In the event of litigation concerning the mark, the original certificate can be a critical 
piece of evidence - hence the importance of safekeeping. I have retained a copy for Goodwin 
Procter's files. Also enclosed is an English translation of the Certificate, and a brief cover letter from 
our local counsel. 

Please note also that if it is not in use already, the mark should be used on these goods and services in 
Russia within three years from the date of registration (i.e., by May 10, 2009), and used continually 
thereafter. The registration will be vulnerable to cancellation for non-use if it can be shown that the 
mark was not used in Russia over a three-year period at any time. You should retain some 
documentary evidence of your first use of the mark in Russia, and on an annual basis set aside some 
evidence of continued use, in the event that we are ever called upon to prove such use. 



Cleaning preparations for household purposes [Intl. Class 03]; and 



Automatic machines for use in the field of interior maintenance of 
households [Intl. Class 07]. 
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This registration is valid for ten years from the priority filing date, or in this case until May 13, 2015. 
The renewal application must be prepared and filed at the IPO within twelve months of the expiration 
of the registration. I suggest that you mark your calendar for May 13, 2014, as a reminder that the 
renewal application is coming due. 

Each of these maintenance/renewal deadlines has been entered on our computerized docketing 
system, so that reminders can be automatically generated and sent at the appropriate times. 
However, we always recommend that our clients also keep independent records of these important 
dates. We also ask that you keep us informed of changes of address or responsible personnel that 
may occur in the intervening years; we cannot be responsible for any missed filings resulting from 
our inability to reach you in the absence of up-to-date contact information. Please also understand 
that we cannot and will not send any such renewal reminders if ARS-Patent is no longer a client of 
Goodwin Procter at that time. 

We are informed that Russian law does not require the use of trademark symbols, but it is a good 
idea to use the ® symbol in proximity to the mark as it is used in Russia, consistent with how the 
mark is used in the United States, as a means of maintaining the distinctiveness of the mark and 
placing potential infringers on notice of your rights. It is also permissible to use the phrases "Trade 
Mark" or "Registered Trademark" in Russian. For materials that will be distributed elsewhere in 
Europe however, we urge caution regarding use of the ® symbol. Rules vary from country to country 
regarding the proper use and legal significance of the ® symbol, and misuse of the ® symbol in 
certain countries can carry severe penalties, so we generally counsel that the common law symbol ™ 
is safer for materials to be distributed in multiple countries outside the U.S. 

It is also our understanding that under applicable Russian law, ownership of a valid registration 
entitles the registrant to the presumptive exclusive right to use the mark (or to license others to use it) 
in Russia, as well as certain causes of action and remedies (including injunctive relief and seizure of 
infringing goods) against infringers. A registrant may also seek cancellation of conflicting later-filed 
marks at the Patent Agency. The registration is considered prima facie evidence of validity and 
ownership of the mark in a Russian court. 

Of course, it is always important regularly to police and enforce your marks by sending "cease and 
desist" letters and taking other appropriate action against persons using identical or confusingly 
similar marks. To that end, there are several international trademark watching services available, at 
varying levels of price and comprehensiveness; we would be happy to provide more information 
about those services if you are interested. 

In this regard, in the weeks and months to come, you may receive notices and solicitations from 
various entities or organizations purporting to offer services in connection with the maintenance, 
publication or policing of your trademark. Many of these notices will appear to be invoices, and 
some are designed to look like they have come from a government agency. These solicitations are 
phony; the services promised are either non-existent or worthless. You will not be required to pay 
renewal or maintenance fees on this registration for another nine years, and you will never be billed 
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directly by the real Russian Patent Agency. If you receive any such invoice or solicitation, please 
forward it directly to me. 

If ownership of the mark or the name of the registrant should change during the term of the 
registration, certain filings should be made with the Patent Agency. We can assist you with such 
filings if they become necessary. It is also important to notify the Agency if the company changes its 
primary business address during the term of the registration, so please keep us informed if your 
offices relocate. 

As you may know, trademarks can be licensed to others, but appropriate license agreements must be 
used to secure the maximum economic benefits to the company and to protect its ownership rights in 
the mark. You should call us for licensing assistance whenever you intend to let any third party 
manipulate, use or display any of the company's trademarks. Among other things, it is critical in 
licensing a trademark to set quality standards for use of the mark to prevent the dilution of the mark 
through its use in connection with inferior goods. (Indeed, the absence of proper quality control 
provisions in a license can result in the invalidation not only of the license but even of the mark 
itself.) In addition, we are informed that Russian law requires that any license agreements 
concerning a Russian-registered mark must be recorded at the Patent Agency in order to be effective. 
Together with local Russian counsel, we can advise and assist in negotiating, drafting and filing 
documentation that complies with local law and protects the company's trademark rights, and in 
particular the good will and reputation of the mark as a symbol of quality. 

Finally, local counsel have also rendered an invoice in the approximate amount of $455.00 for their 
costs and fees incurred in completing this registration. I shall arrange for Goodwin Procter to pay 
this invoice directly and bill ARS-Patent later for the disbursement. 

If you have any questions regarding any of the foregoing, please do not hesitate to contact me. It has 
been our pleasure to assist you and the company in securing this registration, and I look forward to 
working with you on other trademark and intellectual property matters. 

Very truly yours, 



Robert M. O'Connell, Jr. 
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